{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "da1a2754-e6c0-45ec-8bd5-b080673fb26d",
   "metadata": {},
   "source": [
    "# Guideline Evaluator"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "51074c30-8e39-4f30-8125-f1fedb28c679",
   "metadata": {
    "tags": []
   },
   "source": [
    "This notebook shows how to use `GuidelineEvaluator` to evaluate a question answer system given user specified guidelines."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7f647fa2-f007-4242-8c5b-5dbdbb3ad345",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.evaluation import GuidelineEvaluator\n",
    "from llama_index import ServiceContext\n",
    "from llama_index.llms import OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5726c6cf-f9e8-489a-80cb-3272737d2b6f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "GUIDELINES = [\n",
    "    \"The response should fully answer the query.\",\n",
    "    \"The response should avoid being vague or ambiguous.\",\n",
    "    \"The response should be specific and use statistics or numbers when possible.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ca96df17-1c4e-4474-8579-a27d53ac82b3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "service_context = ServiceContext.from_defaults(llm=OpenAI(model=\"gpt-4\"))\n",
    "\n",
    "evaluators = [\n",
    "    GuidelineEvaluator(service_context=service_context, guidelines=guideline)\n",
    "    for guideline in GUIDELINES\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "07ce9b34-73b2-4be7-a1da-3f9103988c9a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sample_data = {\n",
    "    \"query\": \"Tell me about global warming.\",\n",
    "    \"contexts\": [\n",
    "        \"Global warming refers to the long-term increase in Earth's average surface temperature due to human activities such as the burning of fossil fuels and deforestation.\",\n",
    "        \"It is a major environmental issue with consequences such as rising sea levels, extreme weather events, and disruptions to ecosystems.\",\n",
    "        \"Efforts to combat global warming include reducing carbon emissions, transitioning to renewable energy sources, and promoting sustainable practices.\",\n",
    "    ],\n",
    "    \"response\": \"Global warming is a critical environmental issue caused by human activities that lead to a rise in Earth's temperature. It has various adverse effects on the planet.\",\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "69f66574-87dd-4aec-8d73-efe347b701e6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====\n",
      "Guideline: The response should fully answer the query.\n",
      "Pass: False\n",
      "Feedback: The response does not fully answer the query. While it does provide a brief overview of global warming, it does not delve into the specifics such as the causes, effects, and potential solutions to global warming. The response could be improved by providing more detailed information.\n",
      "=====\n",
      "Guideline: The response should avoid being vague or ambiguous.\n",
      "Pass: False\n",
      "Feedback: The response is too vague and does not provide specific details about global warming. It should include more information about the causes, effects, and potential solutions to global warming.\n",
      "=====\n",
      "Guideline: The response should be specific and use statistics or numbers when possible.\n",
      "Pass: False\n",
      "Feedback: The response, while accurate, is not specific enough and does not include any statistics or numbers. It would be more effective if it included specific examples of human activities that contribute to global warming, as well as specific examples of the adverse effects. Additionally, including data or statistics about the rate of temperature increase or the projected impacts of global warming would make the response more informative and impactful.\n"
     ]
    }
   ],
   "source": [
    "for guideline, evaluator in zip(GUIDELINES, evaluators):\n",
    "    eval_result = evaluator.evaluate(\n",
    "        query=sample_data[\"query\"],\n",
    "        contexts=sample_data[\"contexts\"],\n",
    "        response=sample_data[\"response\"],\n",
    "    )\n",
    "    print(\"=====\")\n",
    "    print(f\"Guideline: {guideline}\")\n",
    "    print(f\"Pass: {eval_result.passing}\")\n",
    "    print(f\"Feedback: {eval_result.feedback}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f13e658-982f-4b20-ba08-e639cd7fe1c4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
